youtube
This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper
Attend the next NDC conference near you:
Subscribe to our YouTube channel and learn every day:
/ @NDC
Follow our Social Media!
#bigdata #architecture #database
Detecting problems as they happen is essential in today's fast-moving world. This talk shows how to build a simple, powerful system for real-time anomaly detection using time-tested statistical methods. We'll use Apache Kafka for streaming data, Apache Flink for processing it in real time, and ARIMA models to detect unusual patterns. Whether it's spotting fraud, monitoring systems, or tracking IoT devices, this solution is flexible and reliable.
We'll start by exploring how Kafka helps collect and manage fast-moving data streams. Then, we'll demonstrate how Flink processes this data in real time and integrates ARIMA-based anomaly detection to uncover events as they occur. We'll dive into the details of how ARIMA works—covering how it models trends, seasonality, and residual noise to identify outliers in time-series data.
We'll also show how Apache Iceberg can be used to store historical data efficiently, enabling retrospective analysis, ongoing model evaluation, and performance improvements over time.
By combining real-time detection with long-term storage, you can build a robust system that evolves as your data grows. This talk includes clear examples and practical
|
2026/01/21
youtube
